A Nonparametric Statistical Approach to Clustering via Mode Identification

In this paper, we develop a mode-based clustering approach applying new optimization techniques to a nonparametric density estimator. A cluster is formed by those sample points that ascend to the same local maximum (mode) of the density function. The path from a point to its associated mode is efficiently solved by an EM-style algorithm, namely, the Modal EM (MEM). This clustering method shares the major advantages of mixture model based clustering. Moreover, it requires no model fitting and ensures that every cluster corresponds to a bump of the density. A hierarchical clustering algorithm is also developed by applying MEM recursively to kernel density estimators with increasing bandwidths. The issue of diagnosing clustering results is investigated. Specifically, a pairwise cluster separability measure is defined using the ridgeline between the density bumps of two clusters. The ridgeline is solved for by the Ridgeline EM (REM) algorithm, an extension of MEM. Based upon this new measure, a cluster merging procedure is developed to guarantee strong separation between clusters. Experiments demonstrate that our clustering approach tends to combine the strengths of mixture-model-based and linkage-based clustering. Tests on both simulated and real data show that the approach is robust in high dimensions and when clusters deviate substantially from Gaussian distributions. Both of these cases pose difficulty for parametric mixture modeling.